@xylar Let me know what you think.

Pro of using this approach: no need to run a separate forward step besides those being run. (coeffs_reconstruct doesn't have a time dimension so I think this stream is only written once but it's worth double checking.) I was also thinking we could set write_coeffs_reconstruct = True in polaris/ocean/mpas_ocean.cfg and run whichever cases we want to have cached.

Con of using this approach: if there are multiple forward steps for a single test case (corresponding to the config file for which write_coeffs_reconstruct = True, all will produce the coeffs_reconstruct stream.

The plan will be to

1. Cache all mesh steps for test cases that support Omega (https://docs.e3sm.org/polaris/main/developers_guide/api.html#cache)
2. Use cached mesh steps in Omega PR and nightly suites
3. Add coeffs_reconstruct.nc files corresponding to each mesh to the input_database with date stamps
4. For any viz steps that support Omega and use reconstructed velocity fields, load coeffs_reconstruct.nc and use reconstruct_variable https://github.com/MPAS-Dev/MPAS-Tools/blob/297a1b0bf08c0d9e191c0b0e1d35ae315ee49617/conda_package/mpas_tools/vector/reconstruct.py#L2-L18

This PR addresses #445